fix(perf): resolve concrete EP for the analyzer instead of ep=None (#931)#941
Merged
Conversation
) In `winml perf` without `--ep`, the EP was never resolved before the build, so the static analyzer ran with `ep=None` and aggregated across all EPs (and logged "analyze_onnx called with ep=None — results will aggregate all EPs"), even though inference runs on a single device's EP. Resolve a concrete device + EP from the request and pass it down to the build: - PerfBenchmark resolves device + EP internally (_resolve_device_ep), at the start of _load_model so an unavailable/invalid combo fails fast before the export/optimize/quantize/compile pipeline runs. BenchmarkConfig keeps only the raw request; the resolved values live on the instance and drive from_pretrained / from_onnx. - _perf_modules derives a concrete EP from the resolved device when none is given (explicit EPs are kept verbatim; downstream stages normalize aliases). - The CLI no longer pre-resolves: it just builds the config / dispatches. Verified end-to-end on a CPU build: analyzer target goes from "None on cpu" to "OpenVINOExecutionProvider on cpu" and the ep=None warning no longer fires. Follow-up #939 tracks folding _perf_modules into PerfBenchmark to unify the two resolution sites.
DingmaomaoBJTU
approved these changes
Jun 23, 2026
DingmaomaoBJTU
left a comment
DingmaomaoBJTU
left a comment
Collaborator
There was a problem hiding this comment.
Clean fix. The EP resolution logic is correct, idempotent, and placed at the right point (before the expensive build pipeline). Tests cover all the important cases: derived EP reaches from_onnx, explicit EP passes verbatim, unavailable combo fails fast. One minor nit: when resolve_eps returns [], _resolved_ep silently stays None — a comment making that invariant explicit would help future readers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #931.
In
winml perfwithout--ep, the EP was never resolved before the build, so the static analyzer ran withep=Noneand aggregated across all EPs (logginganalyze_onnx called with ep=None — results will aggregate all EPs), even though inference runs on a single device's EP.This resolves a concrete device + EP from the request and passes it down to the build:
PerfBenchmarkresolves device + EP internally (_resolve_device_ep), at the start of_load_modelso an unavailable/invalid combo fails fast before the export/optimize/quantize/compile pipeline runs (previously it only surfaced atsession.compile()).BenchmarkConfigkeeps only the raw request; the resolved values live on the instance and drivefrom_pretrained/from_onnx._perf_modulesderives a concrete EP from the resolved device when none is given. Explicit EPs are kept verbatim (downstream stages normalize aliases).perf()no longer pre-resolves — it just builds the config and dispatches.WinMLAutoModelstays permissive:ep=Noneremains a valid library mode (aggregate across EPs). The fix makesperf— which targets one device — always pass an explicit EP, which is what the analyzer warning asked for.Verification
End-to-end A/B on a real CPU build (
--no-skip-build, no--ep):ep=NonewarningNone on cpuOpenVINOExecutionProvider on cpuUnit tests (
tests/unit/commands/test_perf_cli.py,test_perf_module.py, 51 passed) cover: derived concrete EP reachesfrom_onnx; explicit EP passes through verbatim; an unavailable combo raises before the build is kicked off.Follow-up
#939 tracks folding
_perf_modulesintoPerfBenchmarkso the two resolution sites become one.